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(54) Automotive occupant sensing device 

(57) A vehicle occuparrt safety system includes an 
image sensor (414) and processor (318) that provides a 
classification of the occupancy status of the passenger 
seat area to determine whether an airt^ag should be 
deployed. The image processing system includes a 
photodetector array sensor (314) and lens (312) assem- 
bly with image processing electronics to acquire a 
machine vision representation of the passenger seat 
area within a vehicle. The objects in the field of view are 
then discriminated to determine whether a rear-facing 
child carrier is located in the passenger seat, such that 
the passenger-side aiit>ag can be disabled. 
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Description 

BACKGROUND HP TH C INVENTION 

1. Retd of the invention 

The present invention relates to in-vehicle imaging 
and sensing and. more particularly, to a device that 
determines whether a vehicle passenger seat is unoc- 
cupied. Is occupied by a person, or is occupied by a 
child seated In a rear-feeing child carrier. 

2. Description of Relait firi Art 

As the needs and desires for automobile safety 
enhancement features increase, injuries attributable to 
such safety equipment and devices themselves are 
exacerbated. For example, automobile passenger-side 
inflatable restraints (airbags) in automobiles equipped 
with such supplemental safety devices are generally 
deployed upon a sufficiently severe impact of the front 
section of the vehicle. However, it has been found that 
rear-fadng child carriers, i.e., where the restrained child 
faces the passenger seat back of the automobile, may 
be potentially hazardous, and possibly perilous, upon 
deployment of an airbag. Serious damage to the child 
restrained in a typical child can-ier could occur when the 
airbag first deploys, shooting out at speeds up to 200 
mph. As illustrated in Figure 1. the inflating airbag 110 
deploys over the top of the child seat 1 12 and transfers 
a force to the back of the seated child's head. 

In addition to possible injury of the child due to the 
direct contact with the airbag, injuries could also occur 
to a child in a rear-facing child carrier if a deploying air 
bag pushes the child restraint 112 into the passenger 
seat back 1 1 4. Although it will be recognized that a safer 
mode of travel for a child in a child can-ier is in the back 
seat of a vehicle, some vehicles do not have back seats 
(e.g., trucks and small sports cars), while In other vehi- 
cles the back seat may be unable to accommodate a 
child carrier 

Furthermore, in some vehicles which have small 
cab areas, such as compact pick-up trucks and sports 
cars, deploying may cause damage to the vehicle as 
well as injury to the driver If the vehicle windows are 
closed during an impact. In such Instances, if both driver 
and passenger airbags deploy during impact the side 
windows of the vehicle could be shattered and the ear- 
drums of the driver ruptured due to the rapkJ air pres- 
sure increase In the small interior volume. Moreover, the i 
replacement cost of a passenger airbag after deploy- 
ment, which would othenwise not have been necessary 
due to the absence of such a passenger, may be sul>- 
stantial. 

Thus, to increase child safety in a rear-fadng child s 
carrier, as well as to lower the cost of unnecessarily 
deployed airbags. a variety of detection technologies 
have been suggested. For example, manual oven-We 
switches may be installed to allow a driver to disable the 



passenger-side airiaag manually Such devices, how- 
ever, become ineffective in instances where the driver or 
operator simply forgets to turn the switch on or off 
depending upon the existence of a passenger or a child 

5 in a rear-facing child carrier in the automobile passen- 
ger seat. Even such enhancements as dashboard indi- 
cators or automatic reset anangements would not be 
foolproof. If a driver transporting a chfW In a rear-facing 
child carrier makes frequent stops, the requirement that 

10 the driver continually manually reset the switch could be 
cumbersome. 

Other safety-enhancement schemes for occupant 
detection indude radar or ultrasonic technologies. Sen- 
sory devices which detect radar or sound waves may be 

15 installed in the dashboard or in the passenger seat 
itself. However, if the dashboard is blocked or the seat Is 
covered, accurate detection of a passenger in the pas- 
senger seat would be hindered. Moreover, it has been 
found that consumers generally do not like the idea of 

20 "beams and waves" being directed at them. Conse- 
quently, such approaches are typically not preferred, 

SUMiWIARYQFTHF im\/cmti/^m 

?5 Accordingly, an object of the present invention is to 
provide an automotive occupant safety system to 
enhance the safety of occupant restraint systems in 
vehides. A further c^ect of the invention is to provide a 
communication and detection system in which aiitag 
'0 control electronics in a vehicle communicate with the 
present invention to determine the preferred airbag 
deployment depending on the occupancy status of the 
seating area. 

In accordance with these and other objects, an 
5 occupant safety system in accordance with the present 
invention indudes image processing technology in con- 
junction with other sensors, such as seat belt extension, 
vehide speed, door open status, seat weighting, etc., to 
determine the preferred airbag deployment. In preferred 
? embodiments, the image processing system generally 
comprises a phototransistor array sensor and lens 
assembly with image processing electronics to acquire 
a visual representatfon of the passenger seat area. The 
objects in the field of view are then discriminated to 
' determine whether a person or a child in a rear-feeing 
child carrier is present in the passenger seat. 

The photodetector array principally acquires 
images based upon photon reception in the near-infra- 
red and visible light spectrum. To operate in dark or une- 
ven lighting conditions without distracting the driver with 
a visible light source, the present invention directs an 
infrared source at the passenger seat area. The infrared 
emitter is mounted on a circuit board which preferably 
includes a processing element, memory devices that 
contain operational software and feult code information, 
interfaces to the vehide electronic systerre, and a 
power supply. 

Emtxxiiments of the present invention are prefera- 
bly located within the interior of a vehide, and mounted 
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in the roof headliner or overhead console. An occupant 
sensing system in accordance with the present inven- 
tion is located to maximize viewing of the passenger 
seat area and minimize potential viewing obstructions 
such that the passenger-side airbag deployment can be 5 
optimized depending upon certain conditions. 

Other objects and aspects of the invention will 
become apparent to those skilled in the art from the 
detailed description of the invention which is presented 
by way of example and not as a limitation of the present 10 
invention. 

BRIEF DESCRIPTION OF T HE DRAWINGS 

Rgure 1 depicts a typical operation of an inflatable 15 
restraint safety system in conjunction with a child car- 
rier. 

Figure 2 shows a perspective view of an automotive 
occupant sensor system in accordance with a preferred 
embodiment of the present invention. 20 

Figure 3 shows one aspect of the sensor system in 
accordance with an embodiment of the invention. 

Figure 4 shows an exploded view of the sensor 
anrangement of the automotive occupant sensor system 
embodiment of Figure 3. 25 

Figure 5 shows a lens arrangement in accordance 
with an emtxxiiment of the present invention. 

Figure 6 is a flow diagram of the occupant sensor 
image processing scheme in accordance with an 
embodiment of the invention. 30 

Figure 7 is a block diagram of the occupant sensor 
according to a preferred embodiment of the present 
invention. 

Figure 8 is a flow diagram of the general operation 
of the automotive occupant sensor system in accord- 3S 
ance with an embodiment of the invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

40 

The following is a description of the best presently 
contemplated mode of carrying out the invention. In the 
accompanying drawings, like numerals designate like 
parts of the se\/eral figures. This description is made for 
the purpose of illustrating the genera! principles of 45 
embodiments of the invention and should not be taken 
in a limiting sense. The scope of the invention is deter- 
mined by reference to the accompanying claims. 

An automotive occupant sensor system 200 In 
accordance with a preferred embodiment of the present so 
invention is shown generally in Figure 2. Embodiments 
of the present invention are directed to the communica- 
tion and coordination with a vehicle airbag electronic 
control module 212 to periodically sense the presence 
or absence of a forward-feeing human being in the front 55 
outboard, passenger seat of the vehicle. The existence 
or absence of a passenger then allows the occupant 
sensor system to make a determination whether to ena- 
ble, inhibit, or otherwise modify the initiation or deploy- 



ment of a passenger-side airbag during a collision or 
other impact. 

Preferred embodiments of the present invention 
perform a restraint system sensing function to control 
the deployment of the passenger-side airbag such that 
it is deployed on demand only if a forward-facing human 
being is present in the passenger seat. More particu- 
larly, embodiments of the invention are directed to differ- 
entiating between a child seated in a rear-facing child 
carrier and a fonward-facing human in the passenger 
seat. It will be recognized that in alternate embodiments 
of the present invention, deployment of an air bag may 
be disabled only upon detection of a child sitting in a 
rear-facing child carrier in the passenger seat of the 
vehicle. In accordance with such embodiments, the air- 
bag would deploy upon impact or collision if any for- 
ward-facing person or object is disposed in the sensor 
viewing area. e.g.. the passenger seat. It also will be 
recognized that in alternate embodiments of the present 
invention, the relative size and position of occupants will 
be determined by the occupant sensing system, with 
the information used to optimize the initiation, rate, and 
inflated volume of the deployed airbag. 

As shown in Figure 2, preferred embodiments of 
the present invention 200 are located in the upper con- 
sole 202 of the vehicle, above the driver's line of vision 
as well as above the passenger seat 204. Mounting 
embodiments of the automotive occupant sensor sys- M 
tem in the overhead console 202 is generally: preferred ?v 
for supporting embodiments of the present Invention to - 
allow the features of the most of the vehicle interior 
seating area to be seen. The configuration of the sensor 
embodiments must be such that arms and objects do 
not block the viewing area. Accordingly, as indicated in 
Figure 2, embodiments of the present invention are % 
preferably mounted at or near the center console area 
202. rear-view mirror (not shown), or pillars 206. 208, 
and 210. near the headliner on either the passenger or 
drivers side of the vehicle. It will be recognized that 
other mounting configurations may be used wherever 
the sensor emtxxfiments are able to adequately view 
the interior of the vehicle, and accurately distinguish 
passengers and/or objects. 

The general operation of prefenred emfcxxJiments of 
the invention is described with reference to Rgure 3. An 
image 310 (shown as a woman holding a book) is 
viewed through a lens 312. The lens 312 is provided to 
evenly focus the image 310 across a photodetector 
array 314. The photodetector 314 may include analog 
and/or digital processing capabilities. In preferred 
embodiments, the interface electronics or image 
processing algorithms are incorporated into the sensor 
314 itself or the sensor electronics, e.g., circuit board 
410 of Figure 4. 

Similarly, an analog-to-digital converter 316 cou- 
pled to the output of the photodetector array 314 may be 
incorporated within the sensor or photodetector array 
314. Tlie output of the A/D converter 316 is input into a 
general purpose processor 318 which performs the 



3 



BNSDOCID: <EP 0734909A2_L> 



5 



EP0734 909A2 



6 



image processing. Any image processing that was not 
accomplished by the photodetector array electronics 
would be performed by the processor 318, which ulti- 
mately determines whether the image of the passenger 
seat area represents a child seated in a rear-facing child 
carrier. 

More particularly, some of the components of an 
embodiment of the occupant sensor module 200 are 
shown in Rgure 4. In the illustrated embodiment, multi- 
ple electronic components are mounted on a circuit 
board 410. A more detailed description of the circuit 
board electronics is discussed below with reference to 
Figure 7. In a preferred embodiment, the circuit board Is 
made with integral interconnections of af lexible material 
allowing it to conform to restricted volume limitations for 
the system package. Preferably, the flexible circuit 
board 410. the sensor array/lens mount/lens assembly 
414. and the IR source 416 are manufactured as a sin- 
gle unit, so that the unit can be quickly and securely 
attached into a housing 418 using the appropriate fas- 
teners. The housing may be formed of plastic, metal, or 
conrposite material, which may be integral with the 
mounting location, or may be removable or othenwise 
portable. 

In preferred embodiments,.the housing and elec- 
tronic sensor module 200 is securely and firmly 
mounted to the vehicle interior. The circuit board 410 is 
preferably installed in the housing 418 and coupled to 
the power source and vehicle electronics via a connec- 
tor 412 which typically varies according to the vehicle 
manufacturer's specifications. Thus, for example, the 
power supply for the sensor 414 and all of the electron- 
ics on circuit board 41 0 can be shared with other vehicle 
control electronics. Preferably, the sensor 414 and cir- 
cuit board 410 arrangement includes a discrete or other 
method off fault indication to communicate if a con-ect 
decision cannot be made by the system due. for exam- 
ple, to a blocked view or an internal fault. Other connec- 
tion lines may include an enable/inhibit line to indicate to 
the airbag electronics 212 (Figure 2) whether a fonward 
facing passenger has been delected. The discrete indi- 
cation electronics may Include industry standard inter- 
faces to provide connection to other processing 
electronics within the vehicle. 

The sensor assembly 41 4 includes a lens 420. lens - 
mount 422, and the sensor itself 510. Although in pre- 
tended embodiments, the packaging of these compo- 
nents may be coupled integrally in the manufacturing 
process, the functions performed by each conponent 
are generally separate. For example, for low-cost, high- f 
volume manufacturing, the sensor/lens/mounting 
assembly may be produced together in a single unit, 
e.g., an injection-molded acrylic embedded with the 
necessary contacts for electrical connections. By pro- 
ducing the lens and lens mounting integrally, manufac- s 
turing costs can be reduced by minimizing assembly 
variances to conform to required tolerances. Such a sin- 
gle unit configuration allows consistent placement and 
alignment of the individual components. It will be recog- 



nized, however, that a variety of other materials and 
substances may be used in the manufacture of the sen- 
sor, lens, and lens mounting assembly in a single unit, 
or as separable conponents. 

5 According to embodiments of the invention, an 
infrared source 416 provides illumination of the vehicle 
interior preferably by infrared LEDs, a bulb with proper 
filtering, or by another lighting device which can ade- 
quately illuminate the entire sensor field of view, ena- 

10 bling image acquisition in the absence of visible light. 
The sensor 414 is sensitive to illumination in the near- 
infrared band which is generally not visible to humans. 
An infrared bandpass filter may be placed in front of the 
lens 420 or be integrated with the system housing 418 

15 to limit the operation of the sensor to the near-bed 
region and to obscure the internal components of the 
system from vehicle occupants. The sensor 414 is pref- 
erably a image based sensor and ultimately translates 
captured light energy into a gray scale image. For exam- 

30 pie. the sensor an^ay and support electronics may be 
used to transmit a 64 x 64 pixel image, with each pixel 
being represented by one of a possible 256 gray shades 
to the processing components of the sensor module cir- 
cuit board 410. 

?5 Figure 5 shows a more detailed side view of the 
sensor assembly 414 comprising a lens 420 mounted to 
a photodetector array/circuit board 510. A cable 512 
couples the sensor to the circuit board 410 (shown in 
Figure 4). In preferred embodiments, the circuit board 

0 410. photodetector an-ay/circuit board 510 and the inter- 
connection cable 512 are manufactured in a flex circuit 
board to simplify interconnections and assembly labor. 
More particularly, the sensor 414 includes a photode- 
tector array 510 which is preferably formed of multiple 

5 silicon phototransistors. The photodetector array 510 
acts as a light gathering tool which may include image 
processing capabilities of its own. The photodetector 
array 510 (shown in Figure 5) is preferably of conven- 
tional CMOS design and therefore could include the 

> functions of other (normally separate) electronic com- 
ponents. 

A processor 724 (shown in Figure 7) included on 
the circuit board 410 analyzes the raw analog or digital 
image, and performs a histogram equalization to opti- 
mize the contrast between black and white. This is per- 
formed by shifting the whites whiter and blacks darker, 
and distributing gray shades evenly from white to black! 
Ukely edges are ttien distinguished, and the angles of 
tile edges are determined such that comparison may be 
made witti stored templates describing ttie classifica- 
tions of different objects, e.g.. child carrier seats and 
enpty passenger seats. Thus, since child seats will fall 
into a variety of different categories based on tiieir gen- 
eral design, tiie distinguished refined edges of tfie 
image are compared to the different templates to deter- 
mine if any match the characteristics of ttie current 
acquired image. When a match is found between ttie 
cun-ent image and a template, a degree of confidence is 
assigned to ttie match. 
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Since movement or shifting of child restraints or an 
empty passenger seat will be limited, changes between 
consecutive captured images that represent dynamic 
motion (once the effects of varying lighting conditions 
are eliminated) can be used to gain confidence in the 5 
detection of human occupants. In addition, the history of 
prior classifications and their associated confidence val- 
ues can be constantly analyzed to improve the reliability 
of the current classification decision generated by the 
occupant sensor. Thus, if the overall confidence, or cal- 10 
culated accuracy, is high based upon the conclusion 
that the sensed image is, for example, a rear-facing 
child carrier, the processor will communicate this status 
to the airbag control electronics to inhibit the deploy- 
ment of the passenger-side airt^ag. 7 5 

Figure 6 describes steps for processing and classi- 
fying images viewed by the sensor according to a pre- 
ferred embodiment of the present invention. A sensed 
input image 610 is acquired by the photodetector array. 
The raw sensed image 610 is then input into a preproc- 20 
essor 612 which applies certain intensity mapping func- 
tions to provide enhanced details, edges, and normalize 
intensity which minimizes threshold adaptation in later 
stages. The preprocessed image 612 then becomes the 
sub-image space 620 which is input into a segmentor 2S 
622. The segmentor 622 includes a spoke filter which 
can identify regions in an image which include parts of 
objects, passengers, and potential "sunspots" which 
could lead to incon-ect identification of false edges. The 
filter allows differentiation and clarification of unnatural 30 
edges due to things like shadows and sunlight stream- 
ing across the image space. 

The output of the segmentor 622 becomes an 
enhanced image 630. The enhanced image is then 
input into an interest point locator (IPL) 632. Preferably. 3S 
the IPL 632 is a Sobel edge operator which examines 
the image and identifies the probable essential edges of 
the image. The edge operator 632 also identifies the 
angles of the edges based upon the positions of the 
edge pixels of adjacent neighbors. Thus, the IPL 632 40 
indicates whether the edges are vertical, horizontal or at 
45** angles relative to the enhanced image to classify 
the image in accordance with stored data representing 
typical shapes and configurations of conventional child 
carriers, empty seats, and passengers. 4S 

In preferred embodiments, typical shapes and con- 
figurations for comparison are included in a template or 
reference library indicating the various types of objects 
or images to be differentiated and distinguished. For 
example, in preferred embodiments, the template or ref- so 
erence library is preferably stored in at least one mem- 
ory device included in the sensor electronics on the 
circuit board 410 (Figure 4). It will be recognized that a 
variety of filters may be implemented for the IPL 632. 
Both software and hardware versions of edge detection ss 
filters have been examined, and may be incorporated 
into embodiments of the invention to maximize the edge 
detection, distinction, and differentiation. 



As shown in Figure 6. the output of the edge opera- 
tor which performs the functions of the IPL 632 along 
with the output of the spoke filter becomes the feature 
vector space 640. The feature vector space 640 is then 
input into the feature correlator 642 which tries to match 
the feature vector space 640 to the different stored tem- 
plates, or reference library, which contains data describ- 
ing idealized child carrier seat, empty seat, and 
passenger classifications. More particularly, the refer- 
ence library includes specific descriptive information on 
image related characteristics of each object classifica- 
tion. Consequently, groups of basic characteristics 
describing the edge shapes of such objects can be for- 
mulated for comparison with the sensed images. 

In preferred embodiments of the invention, the fea- 
ture correlator 642 is capable of translating the refer- 
ence library of data by shifting or rotating the templates 
in space to account for inaccuracies in sensor alignment 
within the vehicle, the range of possible passenger seat 
positions, improper installation of child carrier seats. 
arxJ random movement of passengers. The feature cor- 
relator 642 is further capable of examining only particu- 
lar segments of the templates to correlate the segments 
individually or as a whole with the feature vector space 
640. Thus, once the acquired sensed image has been 
distinguished with highlighted or differentiated edges, 
the image can be matched with the data stored in the 
reference library. Because a family of child carriers has 
been stored in preferred embodiments, the sensor proc- 
essor can adjust for movement or slight rotation of the 
person or object in the vehicle front passenger seat. 

Following the correlation, the output of the feature 
correlator 642 is directed into the classifier feature vec- 
tor space 650. The classifier feature vector space 650 
provides a higher level of the decision-making process, 
and can further distinguish tiie sensed image in light of 
variations or inconsistencies in lens angle, position, and 
focus. The feature vector space 650 is then input into 
tiie classifier 652. which provides an enhanced function 
of classifying the correlated image. Unlike the feature 
correlator 642 in which the feature vector space 640 is 
compared to the different stored templates, the classi- 
fier 652 utilizes conf idences assigned to each object to 
determine the most probable image classification. 

The classifier 652 also refers to other external 
inputs, such as history information or apriori informa- 
tion. For example, after observing five images in the 
passenger seat of a vehicle, the sensor may determine 
with a 99% confidence that a child carrier seat is being 
observed. For the next image, it may also determine 
with 77% confidence that the seat is empty. Using the 
information that a rear facing child carrier was most 
likely observed in the past five images, a single empty 
seat indication may be questioned until more consistent 
and definite verification is achieved. Thus, the classifier 
652 is capable of adapting its observations and determi- 
nations according to different vehicle seat positions and 
viewing perspectives. 
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The output of the classifier 652 is ultirnately com- 
municated to the vehicle electronics and possibly the 
driver at the decision output 660. At this point, the com- 
munication of a final decision indicating the class of the 
image is made. In preferred embodiments, the confi- 5 
dence level. In terms of percentages, is translated Into 
an airbag enable/disable decision which is communi- 
cated to the vehicle airbag control electronics. Conse- 
quently, upon a determination, for example, that an 
Image is a rear-facing child earner seat, the classifier w 
652 effectively disables the deployment of the passen- 
ger-side airbag upon an Impact. 

Rgure 7 is a detailed description of preferred 
embodiments of the circuit board 410 shown in Rgure 4. 
A sensor 710 is coupled to an application specific inte- 15 
grated circuit (ASIC) 708 that implements numerous 
functions. Part of the ASIC 708, the sensor timing and 
control unit 712, is provided with several lines connect- 
ing the ASIC 708 to the sensor 710. The timing and con- 
trol unit 712 provides horizontal and vertical controls 20 
necessary to acquire an image from the sensor pixel by 
pixel, and format the image into random access remory 
(RAM) 734. The image data is then combined to repre- 
sent a complete image. 

A clock 714 is used to log time for a real time clock 25 
and diagnostic time logger (RTC/DTL) 716. The 
RTC/DTL 716 provides the real time of events to a fault 
detection and diagnostic tool interface 718. Whenever 
the system is operational, the fault detection circuitry 
718 continuously perfbmis a self-test diagnostic tunc- 30 
tion to verify the proper operation of the system. Prefer- 
ably, the diagnostic tool interface is provided by the 
vehicle manufacturer to diagnose problems. The diag- 
nostic tool interface 718 represents the circuitry which 
accepts commands from the external diagnostic equip- 3s 
ment provided by the vehicle manufacturer to initiate 
self-tests for calibration and communicate tiie deter- 
mined status back to the diagnostic tool. 

An external bus interface 720 is coupled to the foult 
detection and diagnostic tool interface 718. The bus 40 
interface 720 is provides communications to other on- 
board electronics as required by the vehicle manufac- 
turer. For example, this would represent the electronics 
necessary to perform as a SAE J 1850 bus interface, if 
chosen by the vehicle manufacturer as the data commu- 45 
nication interface. It will be recognized, however, that 
other buses or connecting devices may be utilized. 

Two discrete lines extending from the fault detection 
and diagnostic tool interface 718 indicate whether a 
fault has been detected (line 730) and/or whether the so 
airbag should be enabled or disabled (line 732). The 
fault line 730 signifies to tiie vehicle that a self-test prob- 
lem has been diagnosed or that the image is not opera- 
tional. The enable/inhibit line 732 communicates with 
the airbag electronic control module to indicate that the 55 
airbag deployment should be disabled when a rear-fac- 
ing child carrier seat or empty seat condition has been 
detected. 



The microcontroller macro 724 performs image 
processing on the image acquired by the sensor 710. A 
supervisor/watchdog timer 726 checks the system peri- 
odically to ensure that the ASIC 708 is functioning prop- 
erly. In prefen-ed embodiments, tiie watchdog timer 726 
operates only In the key-on position, i.e., when the igni- 
tion is switched on, to reduce necessary battery power. 
A crystal 742 acts as the reference clock for the micro- 
controller 724. An automobile battery provides a nomi- 
nally 12 volt DC power source that is regulated to +5 
volts DC by tiie voltage regulator 740 for use by the 
occupant sensor elect* onics. The regulated +5 volt DC 
is supplied to the watchdog timer 726 as well as an 
EEPROM 736. The EEPROM 736 stores fault code 
information provkied to the vehicle electronics that mon- 
itor status. RAM 734 may be a separate device or 
included in the ASIC 708 if sufficient capacity exists. 

AHhough in preferred embodiments of the inven- 
tion, the ASIC 708 includes the several components as 
discussed above and illusti-ated in Figure 7, it will be 
recognized that the various components, e.g., tfie clock, 
interfaces, microcontroller, may be coupled together as 
discrete components. 

Rgure 8 is a flow diagram of the operation of a pre- 
ferred embodiment of the present invention. Typically, 
three scenarios are possible. In tiie first scenario, tiie 
front passenger seat is empty 810. Second, tiiere is a 
rear-facing child carrier seat in the front passenger seat 
area 812. In the third scenario 814, all other seating 
an-angements are represented, e.g., forward facing pas- 
sengers, child seats, and other objects. In general, if 
scenarios 810 and 812 are obsen/ed. tiie passenger 
side airt)ag will be disabled. In any otiier expected case 
814, tiie airbag would be enabled. 

In operation, when the driver of the vehicle turns on 
tiie ignition (Step 816). the decision-making process 
begins; at which time, the sensor initially acquires the 
image. The image will be in one of tiie ttiree potential 
scenarios 810. 812, or 814. Sensor embodiments of tiie 
present invention then interpret and classify (step 820) 
tiie image according to the process descn'bed with ref- 
erence to Figure 6. Steps 81 8 through 822 are repeated 
as rapidly as possible, tiie duration of each frame being 
a function of the software performed and the processing 
capability of the occupant sensor. 

Based upon the classification result of tiie particular 
frame or image, and considering any other inputs or 
additional stored informatfon, it is determined whether 
tiie airiDag should be enabled or disabled. Additional 
information may include inputs from other types of sen- 
sors. For example, dashboard-mounted ultrasonic sen- 
sors, a seat pressure sensor, or a seati3elt latch-sensing 
switch, among others, could be integrated. In addition, 
otiier inputs may include tiie past history of tiie most 
recent 5-10 decisions, as well as tiie confidence factors 
which may be associated with the eariier classifications. 
Accordingly, based on the information obtained and 
input, a decision of the cun-ent status of tiie passenger 
seat area can be made. The preferred embodiments of 
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the present invention then communicate the decision 
whether to enable or disable airbag deployment to the 
vehicle airbag electronics (212 in Figure 2). 

Concurrent with the image processing, preferred 
embodiments of the present invention include a self- s 
checking routine to verify continuous proper and reliable 
operation. The self-tests may include electronics 
checks, or basic Image collection verification in which 
periodic checks of certain background features, or anal- 
ysis of a histogram of pixel intensity is performed. io 

In accordance with embodiments of the present 
Invention, the preferred mounting location is in the over- 
head console of the vehicle interior headliner. Depend- 
ing upon the vehicle configuration, alternate mounting 
locations may be preferable for more advantageous is 
sensor viewing or for security-related concerns. As indi- 
cated in Figure 2, pillars A, B, and C. 206. 208. and 210 
respectively on either the passenger or driver's side of 
the vehicle, may be used for mounting the sensing mod- 
ule 200. In addition, it will be recognized that other 20 
arrangements may be implemented. For example, sen- 
sor embodiments may be included within the vehicle's 
dome light structure, rear- view mirror or sun visors. 

It will be recognized that embodiments of the Inven- 
tion may be applied to other image identification pur- 2S 
poses. For example, an alternate embodiment of the 
invention may be used to sense the presence of an 
intruder or an intruder's arm reaching into the vehicle. 
Such applications could be particularly useful in certain 
types of vehicles, such as convertibles or trucks. 30 
Because sensor embodiments for use in a security sys- 
tem preferably provide for a wide-angle view of the 
entire Interior volume of the vehicle, more exact and 
specific image discrimination would most likely not be 
necessary. Thus, image identification with reduced sen- 3S 
sitivity could be implemented to avoid inadvertent 
detection of shadows or other untargeted movement. 

This detailed description is set forth only for pur- 
poses of illustrating examples of the present invention 
and should not be considered to limit the scope thereof 4o 
in any way Clearly numerous additions, substitutions, 
and other modifications can be made to the invention 
without departing from the scope of the invention which 
is defined in the appended claims and equivalents 
thereof. 45 

It should be noted that the objects and advantages 
of the invention may be attained by means of any com- 
patible combination(s) particularly pointed out in the 
items of the following summary of the invention and the 
appended claims. so 

SUMMARY OF INVENTION 

1. A vehicle sensor system for sensing the pres- 
ence or absence of an object in the interior of a ss 
vehicle and classifying an image (310) of the object, 
wherein the object is associated with reflected light 
energy, comprising: 



image classification information representing 
potential images and features of the object; 

storage means for storing the image classifica- 
tion infbrmation; 

an image sensor (414) for viewing the interior 
of the vehicle, the image sensor including: 

a tens (312. 420) for viewing and focusing 
the Image of the object, and 

a photodetector array (314) for acquiring 
the image of the object, and for detecting 
the reflected light energy associated with 
the object, further wherein the reflected 
light energy is represented as raw analog 
image signals; 

an analog-to-digital converter (316) for receiv- 
ing the raw analog image signals, and convert- 
ing the raw analog image signals into digital 
data representing an outline of the image of the 
object; and 

a processor (318. 724) for analyzing the digital 
data and for distinguishing the outline of the 
image represented by the digital data, wherein 
the processor includes 

comparison means (642) for comparing 
the distinguished outline of the image of 
the object with the stored image classifica- 
tion information, wherein if the outline of 
the image of the occupant matches one of 
the stored potential images and features, 
the processor (318. 724) determines that 
the outline of the Image represents the 
stored potential image. 

2. The vehicle sensor system wherein the photode- 
tector array (314) comprises a plurality of pho- 
totransistors. 

3. The vehicle sensor system wherein the vehicle is 
equipped with a passenger seat (114) carrying a 
rear-facing child carrier (112) and a passenger-side 
airbag (212), further wherein a plurality of potential 
images of rear-facing child carriers Is stored in the 
storage means, such that if the processor deter- 
mines that the outline of the image of the object in 
the passenger seat matches one of the potential 
images of rear-facing child carriers, the processor 
causes the passenger-side airbag to be disabled. 

4. The vehicle sensor system further conrtprising 
illumination means (416) for illuminating the interior 
of the vehicle. 

5. The vehicle sensor system wherein the storage 
means, the image sensor (414), the analog-to-dig- 
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ital converter (316), and the processor (724) are 
coupled together on a circuit board (510). 
6. A safety system for enhancing the safety of a 
passenger seated in a passenger seat of a vehicle 
having a passenger-side airlaag controlled by air- s 
bag electronics (212). the safety system for disa- 
bling the passenger-side airbag if an image (310) 
representing a child seated in a rear-fadng child 
carrier seat (112) is identified, the system conpris- 
»ng: 10 

a storage device for storing templates of data 
representing a plurality of seating an-ange- 
ments in the vehicle, wherein at least one of the 
templates defines a child seated in a rear-fac- 75 
ing child carrier seat; 

a sensor (414) for viewing the passenger seat 
of the vehicle, the sensor including: 

20 

a lens (312. 420) for focusing the image of 
the passenger, the passenger having 
associated reflected light energy, 

a photodetector array (314) for detecting 25 
and measuring the reflected light energy of 
the passenger, the photodetector array 
providing output image signals: and 

a processor (318) for comparing the output 3o 
image signals with the stored templates of data 
to identify whether the output image signals 
correlate with one of the plurality of seating 
an-angements represented by the stored tem- 
plates, wherein if a child seated in a rear-facing 3s 
child carrier seat is identified, the processor 
communicates with the aiibag electronics to 
disable the passenger-side alrt>ag. 

7. The safety system wherein the processor com- 4o 
prises: 

means for identifying weak and false edges 

(622) of the output image signals; 

means for segregating (630) the weak and 45 

false edges from the output image signals to 

define an enhanced image; 

means for filtering (650) the enhanced Image to 

disregard false edges and weak edges; and 

means for nrKxIifying the identification of the so 

passenger according to the filtered enhanced 

image. 

8. A method of classifying an image observed by a 
sensor, the method comprising the steps of: 55 

storing classification data representing conven- 
tional configurations of tiie observed image in a 
reference library; 



acquiring an input image of the observed 
image (610); 

distinguishing details and edges of the input 
image (612); 

identifying potential false edges of the input 
image to differentiate unnatural edges 
obstructing tfie input image (622); 
segregating the false edges from the input 
image to define an enhanced image (630); 
determining angles of the edges of ttie 
enhanced image to identify high-probability 
edges and weak edges (632); 
filtering the enhanced image to disregard ^Ise 
edges and weak edges, the filtered enhanced 
image defining a feature vector space (632. 
650); 

correlating the feature vector space with tiie 
stored classification data in fne reference 
library (642. 652); and 

classifying the correlated feature vector space 
according to a match between the feature vec- 
tor space and the reference library (660). 

9. The method further comprising the steps of: 

performing all of the steps of claim 15 multiple 
times; 

storing the classified correlated feature vector 
space values as history data; 

comparing the most recentiy classified corre- 
lated feature vector space with the history data; 
and 

modifying the classified correlated feature vec- 
tor space according to the comparison with tiie 
history data 

10. The method wherein the image observed by the 
sensor represents an occupant seated in the pas- 
senger seat of a vehicle having a passenger-side 
aiitag which is deployed by an airbag electronics 
system (212), and furtiier comprising tiie steps of: 

classifying the correlated feature vector space 
as a chiU seated in a rear-facing child carrier, 
the classified correlated feature vector space 
defining a classified image (822); and 
communicating to the airt^ag electronics sys- 
tem to disable passenger-side airbag (824). 

11. A method of identifying an image in a passen- 
ger seat of a vehicle equipped with a passenger- 
side airbag. tiie method comprising the steps of: 

storing classification data representing at least 
three possible seating arrangements of the 
image, the at least three possible seating 
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arrangements including an empty passenger 
seat (810). a child seated in a rear-facing child 
carrier in the passenger seat (812), and a for- 
ward-facing occupant (814), the classification 
data being stored in a reference library; 
observing the image (610), wherein the image 
is observed for a plurality of frames defined by 
predetermined periods of time; 
acquiring the plurality of frames of the 
observed image (818); 

interpreting the acquired frames of the image 
(820) by 



distinguishing details and 
observed image (612). 



edges of the 



segregating weak and false edges of the 
observed image (630). and 

filtering the weak and false edges from the 
observed Image (632. 650); 

comparing the filtered observed image with the 
stored classification data to determine whether 
the filtered observed image matches one of the 
at least three possible seating arrangements 
(642. 652): and 

classifying the filtered observed image within 
one of the at least three possible seating 
arrangements (660), wherein 

If the passenger seat is determined to be 
empty, the passenger-side airbag is disa- 
bled. 

if a child seated in a rear-facing child car- 
rier is determined to be in the passenger 
seat, the passenger-side airbag is disa- 
bled, and 

If a forward-facing occupant Is determined 
to be seated in the passenger seat, the 
passenger-side airbag is enabled. 



12. The method further comprising the steps of: 45 2. 

storing a plurality of previous frames of the 
observed image as history data; 

3. 

comparing a most recent classified image with so 
the history data; and 

modifying the classified image according to the 
history data. 
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Claims 

1. A vehicle sensor system for sensing the presence 
or absence of an object in the interior of a vehicle 



55 



and classifying an image (310) of the object, 
wherein the object is associated with reflected light 
energy, comprising: 

image classification information representing 
potential images and features of the object; 

storage means for storing the image classifica- 
tion information; 

an image sensor (414) for viewing the interior 
of the vehicle, the image sensor including: 

a lens (312, 420) for viewing and focusing 
the image of the object, and 

a photodetector array (314) for acquiring 
the image of the object, and for detecting 
the reflected light energy associated with 
the object, further wherein the reflected 
light energy is represented as raw analog 
image signals; 

an analog-to-digltal converter (316) for receiv- 
ing the raw analog image signals, and convert- 
ing the raw analog image signals into digital 
data representing an outline of the image of the 
object; and 

a processor (318. 724) for analyzing the digital 
data and for distinguishing the outline of the 
image represented by the digital data, wherein 
the processor includes 

comparison means (642) for comparing 
the distinguished outline of the image of 
the object with the stored image classifica- 
tion information, wherein if the outline of 
the image of the occupant matches one of 
the stored potential images and features, 
the processor (318. 724) determines that 
the outline of the Image represents the 
stored potential image. 

The vehicle sensor system off claim 1, wherein the 
photodetector an^ay (314) comprises a plurality of 
phototransistors. 

The vehicle sensor system of claim 1. wherein the 
vehicle is equipped with a passenger seat (114) 
carrying a rear-facing child carrier (112) and a pas- 
senger-side airbag (212). further wherein a plurality 
of potential images of rear-facing child carriers Is 
stored in the storage means, such that if the proc- 
essor determines that the outline of the Image of 
the object In the passenger seat matches one of the 
potential images of rear-facing child carriers, the 
processor causes the passenger-side airbag to be 
disabled. 
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4. The vehicle sensor system of claim 1 , further com- 
prising illumination means (416) for illuminating the 
interior of the vehicle. 

5- The vehicle sensor system of claim 1 . wherein the s 
storage means, the image sensor (414), the ana- 
log-to-digital converter (316). and the processor 
(724) are coupled together on a circuit board (510). 

6. A safety system for enhandng the safety of a pas- io 
senger seated in a passenger seat of a vehicle hav- 
ing a passenger-side airbag controlled by airbag 
electronics (212). the safety system for disabling 
the passenger-side airbag if an image (310) repre- 
senting a child seated in a rear-fadng child carrier is 
seat (1 12) is Identified, the system comprising: 

a storage device for storing templates of data 
representing a plurality of seating arrange- 
ments in the vehicle, wherein at least one of the so 
templates defines a child seated in a rear-fac- 
ing child carrier seat; 

a sensor (414) for viewing the passenger seat 
of the vehicle, the sensor including: 25 

a lens (312. 420) for focusing the image of 
the passenger, the passenger having 
associated reflected light energy. 

30 

a photodetector array (314) for detecting 
and measuring the reflected light energy of 
the passenger, the photodetector anay 
providing output image signals; and 

35 

a processor (318) for comparing the output 
image signals with the stored templates of data 
to Identify whether the output image signals 
correlate with one of the plurality of seating 
an-angements represented by the stored tern- 40 
plates, wherein if a child seated in a rear-facing 
child carrier seat is Identified the processor 
communicates with the airbag electronics to 
disable the passenger-side airbag. 

45 

. The safety system of claim 6. wherein the proces- 
sor comprises: 

means for identifying weak and false edges 
(622) of the output image signals; so 

means for segregating (630) the weak and 
false edges from the output image signals to 
define an enhanced image: 

55 

means for filtering (650) the enhanced image to 
disregard false edges and weak edges; and 



means for modifying the identification of the 
passenger according to the ffltered enhanced 
image. 

8. A method of classifying an image observed by a 
sensor, the method conprising the steps of: 

storing classification data representing conven- 
tional configurations off the observed image in a 
reference library; 

acquiring an input image of the observed 
image (610); 

distinguishing details and edges of the input 
image (612); 

identifying potential false edges of the input 
image to differentiate unnatural edges 
obstructing the input image (622); 

segregating the false edges from the input 
image to define an enhanced image (630); 

determining angles of the edges of the 
enhanced image to identify high-probability 
edges and weak edges (632); 

filtering the enhanced image to disregard false 
edges and weak edges, the filtered enhanced 
image defining a feature vector space (632, 
650); 

correlating the feature vector space with the 
stored classification data In the reference 
library (642. 652): and 

classifying the correlated feature vector space 
according to a match between the feature vec- 
tor space and the reference library (660). 

9. The method of claim 8, further comprising the steps 



performing all of the steps of claim 15 multiple 
times; 

storing the classified con^elated feature vector 
space values as history data; 

comparing the most recently classified con-e- 
lated feature vector space with the history data; 
and 

modifying the classified con-elated feature vec- 
tor space according to the comparison with the 
history data. 



10 



EP 0734909A2J_> 



19 



EP0 734 909 A2 



20 



10. The method of claim 8. wherein the image 
observed by the sensor represents an occupant 
seated In the passenger seat of a vehicle having a 
passenger-side airbag which is deployed by an air- 
bag electronics system (212). and further compris- 5 
Ing the steps of: 

classifying the correlated feature vector space 
as a child seated in a rear-facing child carrier, 
the classified correlated feature vector space 10 
defining a classified image (822); and 

communicating to the airbag electronics sys- 
tem to disable passenger-side airbag (824). 

15 

11. A method of Identifying an image in a passenger 
seat of a vehicle equipped with a passenger-side 
airbag. the method comprising the steps of: 

storing classification data representing at least 20 
three possible seating arrangements of the 
image, the at least three possible seating 
arrangements including an empty passenger 
seat (810). a child seated in a rear-factng child 
carrier in the passenger seat (812), and a for- 25 
ward-facing occupant (814). the classification 
data being stored in a reference library; 

observing the image (610). wherein the image 
is observed for a plurality of frames defined by 30 
predetermined periods of time; 

acquiring the plurality of frames of the 
observed image (818); 

35 

interpreting the acquired frames of the image 
(820) by 

distinguishing details and edges of the 
observed image (612). 40 

segregating weak and false edges of the 
observed image (630). and 

filtering the weak and false edges from the 45 
observed Image (632. 650); 

comparing the filtered observed image with the 
stored classification data to determine whether 
the filtered observed image matches one of the so 
at least three possible seating arrangements 
(642. 652); and 

classifying the filtered observed image within 
one of the at least three possible seating 55 
anangements (660). wherein , 



if the passenger seat is determined to be 
empty, the passenger-side airbag is disa- 
bled. 

if a child seated in a rear-facing child car- 
rier is determined to be In the passenger 
seat, the passenger-side airbag is disa- 
bled, and 

if a forward-feeing occupant is determined 
to be seated in the passenger seat the 
passenger-side airbag Is enabled. 

12, The method of claim 11. further comprising the 
steps of: 

storing a plurality of previous frames of the 
observed image as history data; 

comparing a most recent classified image with 
the history data; and 

modifying the classified image according to the 
history data. 
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